Recent Progress on the Complexity of Solving Markov Decision Processes

The complexity of algorithms for solving Markov Decision Processes (MDPs) with finite state and action spaces has seen renewed interest in recent years. New strongly polynomial bounds have been obtained for some classical algorithms, while others have been shown to have worst case exponential complexity. In addition, new strongly polynomial algorithms have been developed. We survey these results, and identify directions for further work. In the following subsections, we define the model, the two optimality criteria we consider (discounted and average rewards), the classical value iteration, policy iteration algorithms, and how to find an optimal policy via linear programming. In Section 2, we review the literature on the complexity of algorithms for sovling discounted and average-reward problems. Finally, in Section 3 we consider some directions for further work.